Gathering current knowledge about quality evaluation in software product lines

Recently, a number of methods and techniques for assessing the quality of software product lines have been proposed. However, to the best of our knowledge, there is no study which summarizes all the existing evidence about them. This paper presents a systematic review that investigates what methods and techniques have been employed (in the last 10 years) to evaluate the quality of software product lines and how they were employed. A total of 39 research papers have been reviewed from an initial set of 1388 papers. The results show that 25% of the papers reported evaluations at the Design phase of the Domain Engineering phase. The most widely used mechanism for modeling quality attributes was extended feature models and the most evaluated artifact was the base architecture. In addition, the results of the review have identified several research gaps. Specifically, 77% of the papers employed case studies as a “proof of concept” whereas 23% of the papers did not perform any type of validation. Our results are particularly relevant in positioning new research activities and in the selection of quality evaluation methods or techniques that best fit a given purpose.